Method and system for use of hearing prosthesis for linguistic evaluation

ABSTRACT

A method and system to help determine the extent to which a hearing prosthesis recipient is exposed to speech. The hearing prosthesis will log data regarding audio input, optimally in correspondence with times when the hearing prosthesis is in a stimulation-on mode in which the hearing prosthesis is set to stimulate a physiological system of the recipient in accordance with received audio input, so as to facilitate identification of speech to which the recipient is exposed. Further, as the hearing prosthesis itself receives the audio input, the hearing prosthesis may analyze the audio input to determine one or more linguistic characteristics of the audio input, such as a quantity of speech by the recipient and/or by others, and the hearing prosthesis may output data representing the determined one or more linguistic characteristics. Advantageously, the data may then be used to help facilitate rehabilitation of the recipient.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/892,747, entitled “Method and System for Use of Hearing Prosthesisfor Linguistic Evaluation,” filed on May 13, 2013, now U.S. Pat. No.9,814,879. The entire contents of which is incorporated herein byreference herein.

BACKGROUND

Unless otherwise indicated herein, the information described in thissection is not prior art to the claims and is not admitted to be priorart by inclusion in this section.

Various types of hearing prostheses provide people with different typesof hearing loss with the ability to perceive sound. Hearing loss may beconductive, sensorineural, or some combination of both conductive andsensorineural. Conductive hearing loss typically results from adysfunction in any of the mechanisms that ordinarily conduct sound wavesthrough the outer ear, the eardrum, or the bones of the middle ear.Sensorineural hearing loss typically results from a dysfunction in theinner ear, including the cochlea where sound vibrations are convertedinto neural signals, or any other part of the ear, auditory nerve, orbrain that may process the neural signals.

People with some forms of conductive hearing loss may benefit fromhearing prostheses such as hearing aids or vibration-based hearingdevices. A hearing aid, for instance, typically includes a smallmicrophone to receive sound, an amplifier to amplify certain portions ofthe detected sound, and a small speaker to transmit the amplified soundsinto the person's ear. A vibration-based hearing device, on the otherhand, typically includes a small microphone to receive sound and avibration mechanism to apply vibrations corresponding to the detectedsound so as to cause vibrations in the person's inner ear. Examples ofvibration-based hearing devices include bone anchored devices thattransmit vibrations via the skull and acoustic cochlear stimulationdevices that transmit vibrations more directly to the inner ear.

Further, people with certain forms of sensorineural hearing loss maybenefit from hearing prostheses such as cochlear implants and/orauditory brainstem implants. Cochlear implants, for example, include amicrophone to receive sound, a processor to convert the sound to aseries of electrical stimulation signals, and an array of electrodes todeliver the stimulation signals to the implant recipient's cochlea so asto help the recipient perceive sound. Auditory brainstem implants usetechnology similar to cochlear implants, but instead of applyingelectrical stimulation to a person's cochlea, they apply electricalstimulation directly to a person's brain stem, bypassing the cochleaaltogether, still helping the recipient perceive sound.

In addition, some people may benefit from hearing prostheses thatcombine one or more characteristics of the acoustic hearing aids,vibration-based hearing devices, cochlear implants, and auditorybrainstem implants to enable the person to perceive sound.

SUMMARY

A person who suffers from hearing loss may also have difficulty speakingand appreciating speech by others, including in some cases developinglanguage skills in the first place. When such a person receives ahearing prosthesis to help them better perceive sounds, it may thereforebe important for the person to be exposed to a sufficient extent ofspeech (including both speech production and speech reception) so thatthe person can begin to improve their own speaking and to betterappreciate speech by others. For example, once a person receives ahearing prosthesis, it may be important for the person to produce asufficient quantity of speech themselves and to receive a sufficientquantity of speech from others, such as by engaging in a sufficientextent of speech interaction (e.g., conversation) with others. Further,it may be important for the speech exposure to be of sufficient qualityand varying complexity, including for instance a sufficient number ofclear words, sentences, and the like.

This may be the case especially for recipients of cochlear implants andother such prostheses that do not merely amplify received sounds butprovide the recipient with other forms of physiological stimulation tohelp them perceive the received sounds. Exposing such a recipient to agood amount of speech (both received and produced by the recipient) mayhelp the recipient begin to better correlate those physiologicalstimulations with the received sounds and thus improve the recipient'sspeech and appreciation of speech.

By the same token, knowing the extent to which a hearing prosthesisrecipient is exposed to speech may help a clinician such as a speechtherapist, teacher, or parent develop appropriate therapy to helprehabilitate the recipient. For example, given knowledge that therecipient is not speaking very often, is not being exposed to muchspeech of others, or is not engaging in sufficient conversation, aclinician may help the recipient to speak more often or may arrange forfriends and family members of the recipient to speak with the recipientmore often. As another example, given knowledge that the recipient isbeing exposed to a large amount of speech together with background noisesuch as television or music, a clinician may advise turning off thebackground noise sources to help improve the recipient's speechexposure. And as still another example, given knowledge of certaincharacteristics of the recipient's speech or of other speech in therecipient's environment, a clinician may develop strategies (e.g.,activities or the like) to help the recipient engage in and be exposedto other forms of speech, to further rehabilitate the recipient.

Disclosed herein are methods and corresponding systems to help determinethe extent to which a hearing prosthesis recipient is exposed to speech,including for instance the recipient's own speech, speech by others inthe recipient's environment, and speech interaction between therecipient and others. The disclosed methods leverage the fact that, asthe hearing prosthesis is in use by the recipient, the prosthesis itselfreceives audio input that represents the audio environment of therecipient including speech in that environment.

In accordance with the disclosure, the hearing prosthesis will log dataregarding that received audio input, optimally in correspondence withtimes when the hearing prosthesis is set to stimulate a physiologicalsystem of the recipient in accordance with received audio input, so asto facilitate identification of speech to which the recipient isexposed. Further, as the hearing prosthesis itself receives the audioinput, the hearing prosthesis may analyze the audio input to determineone or more linguistic characteristics of the audio input, such as aquantity of speech by the recipient and/or by others, and the hearingprosthesis may output data representing the determined one or morelinguistic characteristics. Advantageously, the data may then be used tohelp facilitate rehabilitation of the recipient as discussed above.

Accordingly, in one respect, disclosed is a method that can beimplemented by a recipient's hearing prosthesis such as one of thosedescribed above for instance. In accordance with the method, the hearingprosthesis receives audio input representing an audio environment of therecipient, and the hearing prosthesis is operable to stimulate aphysiological system of the recipient in accordance with that receivedaudio input. Further, the hearing prosthesis determines, based on thatsame received audio input, one or more linguistic characteristics of theaudio environment, and the hearing prosthesis generates and outputs datarepresenting the one or more determined linguistic characteristics.

In practice, such a hearing prosthesis may have a stimulation mode thatswitches between a stimulation-on mode in which the hearing prosthesisis set to stimulate the physiological system of the recipient inaccordance with received audio input (e.g., to the extent the receivedaudio input would result lead to such stimulation) and a stimulation-offmode in which the hearing prosthesis is set to not stimulate thephysiological system of the recipient in accordance with the receivedaudio input. (For instance, a cochlear implant may have a coil that maybe on to set the implant in a stimulation-on mode or off to set theimplant in a stimulation-off mode.) In that case, the act of generatingthe data representing the determined one or more linguisticcharacteristics may additionally involve basing the data at least inpart on the stimulation mode of the hearing prosthesis. For instance,the hearing prosthesis may include in the data indicia of times when thehearing prosthesis was in the stimulation-on mode, such as indicia thatindicate which determined linguistic characteristics correspond withthose times of stimulation. Or the hearing device may limit thelinguistic characteristic data to be with respect to just audio inputthat resulted in stimulation of the recipient's physiological system.

In another respect, disclosed is a method that can be similarlyimplemented by a recipient's hearing prosthesis. In accordance with thismethod, the hearing prosthesis receives audio input, and the hearingprosthesis is operable to stimulate a physiological system of therecipient in accordance with the received audio input. Further, at timeswhile receiving the audio input, the hearing prosthesis is in astimulation-on mode in which the hearing prosthesis is set to stimulatethe physiological system of the recipient in accordance with thereceived audio input and at other times while receiving the audio inputthe hearing prosthesis is in a stimulation-off mode in which the hearingprosthesis is set to not stimulate the physiological system of therecipient in accordance with the received audio input. Pursuant to themethod, the hearing prosthesis logs data representing the received audioinput in correspondence with only the times when the hearing prosthesisis in the stimulation-on mode. Thus, the hearing prosthesis may limitthe logging to be with respect to the just the audio input received whenthe hearing prosthesis was set to stimulate the recipient'sphysiological system in accordance with the audio input.

In still another respect, disclosed is a method that can be implementedby a hearing assistance device that is worn by or at least partiallyimplanted into a human recipient. In accordance with this method, thehearing assistance device receives audio input representing an audioenvironment of the recipient, and the hearing assistance device isoperable to stimulate a physiological system of the recipient inaccordance with the received audio input. Further, the hearingassistance device records data representing the received audio input andspecifying times when the hearing assistance device was operating in astimulation-on mode in which the hearing assistance device was set tostimulate the physiological system of the recipient in accordance withthe received audio input. Further, the method includes determining basedon the recorded data one or more linguistic characteristics of the audioenvironment, and providing output that represents one or more determinedlinguistic characteristics.

In yet another respect, disclosed is a hearing assistance device (e.g.,a hearing prosthesis such as any of those noted above) configured to beworn by or at least partially implanted in a human recipient and tostimulate a physiological system of the recipient, such as one of thehearing prostheses discussed above for instance. The disclosed deviceincludes one or more microphones for receiving audio input representingan audio environment of the recipient, and the device is configured tostimulate the physiological system of the recipient in accordance withthat received audio input. Further, the device is configured to (i)determine, based on the received audio input, one or more linguisticcharacteristics of the audio environment, (ii) generate datarepresenting the one or more determined linguistic characteristics and(iii) output the data for evaluation. For instance, the device mayinclude a processing unit, data storage, and program instructions storedin the data storage and executable by the processing unit to carry outthe determining, generating, and outputting functions.

And in yet another respect, disclosed is a hearing instrument (e.g., ahearing prosthesis such as any of those discussed above) including oneor more microphone inputs, one or more stimulus outputs, and aprocessing unit for receiving audio input from the one or moremicrophone inputs, generating stimulation signals, and providing thestimulation signals to the one or more stimulation outputs, so as tostimulate a physiological system of a recipient of the hearinginstrument. The processing unit is then further configured to log dataregarding operation of the hearing instrument and regarding anenvironment of a recipient, in correspondence with data indicating whenthe hearing instrument is set to provide the stimulation and when thehearing instrument is not set to provide the stimulation.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description, with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescription throughout by this document, including in this summarysection, is provided by way of example only and therefore should not beviewed as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of an example hearing prosthesis.

FIG. 2 is a block diagram depicting components of an example system.

FIGS. 3-5 are flow charts depicting functions that can be carried out inaccordance with representative methods.

DETAILED DESCRIPTION

Referring to the drawings, as noted above, FIG. 1 is a simplified blockdiagram of an example hearing prosthesis 12 operable in accordance withthe present disclosure. As shown, the example hearing prosthesis 12generally includes one or more microphones (microphone inputs) 14 forreceiving audio input representing an audio environment of theprosthesis recipient, a processing unit 16 having a translation module18 for translating a representation of the received audio input intostimulation signals, and stimulation means (one or more stimulationoutputs) for stimulating the physiological system of the recipient inaccordance with the stimulation signals and thus in accordance with thereceived audio input.

This example hearing prosthesis may represent any of various types ofhearing prosthesis, including but not limited to those discussed above,and the components shown may accordingly take various forms. By way ofexample, if the hearing prosthesis is a hearing aid, the translationmodule 18 may include an amplifier that amplifies the received audioinput, and the stimulation means 20 may include a speaker arranged todeliver the amplified audio into the recipient's ear. As anotherexample, if the hearing prosthesis is a vibration-based hearing device,the translation module 18 may function to generate electricalstimulation signals corresponding with the received audio input, and thestimulation means 20 may include a transducer that delivers vibrationsto the recipient in accordance with those electrical stimulationsignals. And as yet another example, if the hearing prosthesis is acochlear implant, the translation module 18 may similarly generateelectrical signals corresponding with the received audio input, and thestimulation means 20 may include an array of electrodes that deliver thestimulation signals to the recipient's cochlea. Other examples arepossible as well.

In practice, the processing unit 16 may be arranged to operate on adigitized representation of the received audio input as established byanalog-to-digital conversion circuitry in the processing unit,microphone(s) or one or more other components of the prosthesis. Assuch, the processing unit 16 may include data storage (e.g., magnetic,optical or flash storage) 22 for holding a digital bit streamrepresenting the received audio and for holding associated data.Further, the processing unit 16 may include a digital signal processor,and the translation module 18 may be a function of the digital signalprocessor, arranged to analyze the digitized audio and to producecorresponding stimulation signals or associated output. Alternatively oradditionally, the processing unit may include one or more generalpurpose processors (e.g., microprocessors), and the translation module18 may include a set of program instructions stored in the data storage22 and executable by the processor(s) to analyze the digitized audio andto produce the corresponding stimulation signals or associated output.

Further, the processing unit 16 may control and/or track the extent towhich the hearing prosthesis stimulates the physiological system of therecipient. For instance, as noted above, the prosthesis may have astimulation mode that can be switched between a stimulation-on mode anda stimulation-off mode. In the stimulation-on mode, the prosthesis wouldbe set to stimulate the physiological system of the recipient inaccordance with audio input being received by the prosthesis, such as byproviding corresponding stimulation signals when the audio input is ofsufficient amplitude and of particular frequency and to not providestimulation signals if the audio input is threshold low amplitude or isof some other frequency. And in the stimulation-off mode, the prosthesiswould be set to not stimulate the physiological system of the recipientin accordance with the audio input being received by the prosthesis,regardless of the amplitude and frequency of the audio input.

The processing unit may switch between these stimulation modes inaccordance with user input or may determine when a user has switched thehearing prosthesis between these stimulation modes. For instance, theprocessing unit may receive user input directing the hearing prosthesisto switch between stimulation-on mode and stimulation-off mode, and theprocessing unit may responsively set the stimulation mode accordinglyand make note of the current stimulation mode. Alternatively, a user maychange the stimulation mode of the prosthesis (such as by switchingbetween a coil-on mode and coil-off mode), and the processing unit maymake note of that change in stimulation mode. Further, the processingunit may switch the prosthesis between stimulation-on mode andstimulation-off mode from time to time based on a programmed schedule orother trigger events and may similarly make note of the currentstimulation mode.

As further shown, the example hearing prosthesis 12 includes or iscoupled with a user interface system 24 through which the recipient orothers (e.g., a clinician) may control operation of the prosthesis andview various settings and other output of the prosthesis. In practice,for instance, the user interface system 24 may include one or morecomponents internal to or otherwise integrated with the prosthesis.Further, the user interface system 24 may include one or more componentsexternal to the prosthesis, and the prosthesis may include acommunication interface arranged to communicate with those componentsthrough a wireless and/or wired link of any type now known or laterdeveloped.

In a representative arrangement, the user interface system 22 mayinclude one or more user interface components that enable a user tointeract with the hearing prosthesis. As shown by way of example, theuser interface components may include a display screen 26 and/or one ormore input mechanisms 28 such as a touch-sensitive display surface, akeypad, individual buttons, or the like. These user interface componentsmay communicate with the processing unit 16 of the hearing prosthesis inmuch the same way that conventional user interface components interactwith the host processor of a personal computer. Alternatively, the userinterface system 24 may include one or more standalone computing devicessuch as a personal computer, mobile phone, tablet, handheld remotecontrol, or the like, and may further include its own processing unit 30that interacts with the hearing prosthesis and may be arranged to carryout various other functions.

In practice, user interface system 24 may enable the recipient tocontrol the stimulation mode of the hearing prosthesis, such as to turnstimulation functionality on, and off. For instance, at times when therecipient does not wish to have the prosthesis stimulate the recipient'sphysiological system in accordance with received audio input, therecipient may engage a button or other input mechanism of the userinterface system 24 to cause processing unit 16 to set the prosthesis inthe stimulation-off mode. And at times when the recipient wishes to havethe prosthesis stimulate the recipient's physiological system inaccordance with the received audio input, the recipient may engage asimilar mechanism to cause the processing unit 16 to set the prosthesisin the stimulation-on mode. Further, the user interface system 24 mayenable the recipient or others to program the processing unit 16 of theprosthesis so as to schedule automatic switching of the prosthesisbetween the stimulation-on mode and the stimulation-off mode.

In accordance with the present disclosure, as noted above, the examplehearing prosthesis 12 will additionally function to log and output dataregarding the received audio input. In particular, the hearingprosthesis may analyze the received audio input so as to determine oneor more linguistic characteristics in the recipient's audio environmentand may output data representing the determined one or more linguisticcharacteristics. Further, the hearing prosthesis may use its stimulationmode as a basis to generate this data, such as by determining andlogging linguistic characteristics just with respect to the audio inputreceived while the hearing prosthesis is in the stimulation-on mode, orby separately recording (i) linguistic characteristics in the audioinput received at times when the hearing prosthesis was in thestimulation-on and (ii) linguistic characteristics in the audio inputreceived at times when the hearing prosthesis was in the stimulation-offmode.

The hearing prosthesis may then output it logged data from time to timefor external analysis, such as for external determination and reportingof linguistic characteristics in the recipient's audio environment. Forinstance, the user interface system 24 may periodically poll the hearingprosthesis to obtain from the prosthesis the latest linguisticcharacteristics logged by the prosthesis, such as the latest loggedlinguistic characteristics corresponding with stimulation-on mode andthe latest logged linguistic characteristics corresponding withstimulation-off mode. And the user interface system 24 may process thatdata and provide a graphical user interface that depicts a comparison ofthe logged linguistic characteristics (possibly per stimulation mode)over time.

Notably, the audio input that forms the basis for this analysis is thesame audio input that the hearing prosthesis is arranged to receive anduse as a basis to stimulate the physiological system of the recipientwhen the prosthesis is in the stimulation-on mode. Thus, as the hearingprosthesis receives audio input, the hearing prosthesis may not onlytranslate that audio input into stimulation signals to stimulate therecipient's physiological system if the hearing prosthesis is in thestimulation-on mode but may also log data regarding the same receivedaudio output, such as data regarding linguistic characteristics in theaudio input in correlation with the stimulation mode. Further, even attimes when the hearing prosthesis is receiving audio input but is notstimulating the recipient's physiological system (e.g., becausestimulation is turned off or because the audio input amplitude orfrequency is such that the prosthesis is set to not providestimulation), the hearing prosthesis may still log data regarding thatreceived audio input, such as linguistic characteristics in correlationwith the stimulation mode. Any or all of this data may then beclinically relevant and useful in developing therapy and training (e.g.,remediation) for the recipient.

As shown in FIG. 1, the processing unit 16 of the example hearingprosthesis 12 includes a data logging and linguistic analysis (DLLA)module 32 for carrying out some or all of these added functions. ThisDLLA module 32 may be integrated in whole or in part with thetranslation module 18, such as by making use of some of the samecomponents of the hearing prosthesis as the translation module 18.Further, as with the translation module, this DLLA module may beprovided in various forms. For instance, the DLLA module may be providedas a function of a digital signal processor, or as a set of programinstructions stored in data storage and executable by one or moreprocessors to carry out the data logging and linguistic analysisfunctions.

In practice, as the processing unit 16 receives audio input representingthe audio environment of the recipient, the processing unit module mayevaluate the audio input in real-time so as to determine one or morelinguistic characteristics in the audio input.

The “linguistic characteristics” explored here are characteristicsspecifically related to language production and receipt, and thereforemay or may not include more general audio characteristics such asamplitude, frequency, or the like. Examples of linguisticcharacteristics include, among others, (1) a measure of proportion oftime spent by the recipient speaking, (2) a measure of proportion oftime spent by the recipient receiving speech from others, (3) a measureof quantity of words spoken by the recipient, (4) a measure of quantityof sentences spoken by the recipient, (5) a measure of quantity of wordsspoken by one or more people other than the recipient, (6) a measure ofquantity of sentences spoken by one or more people other than therecipient, (7) a measure of quantity of conversational turns by therecipient, (8) a measure of length of utterances by the recipient or byothers, (9) a measure of quantity of phonetic features produced by therecipient, such as voiced vs unvoiced speech sounds, vowels versusconsonants, or a more specific breakdown of consonant articulation suchas plosives, affricatives, fricatives, sibilants, nasal, flap, tap,approximant, lateral, trill, and so forth, including for instance ameasure of rate of syllabic or other speech production and/or a measureof phoneme variations created, (10) a measure of quality of speechexposure, such as presentation level and signal to noise ratio of thespeech, (11) a measure of words spoken by adult versus words spoken bychildren, (12) a measure of quantity of conversations engaged in orinitiated by the recipient, and (13) indications of whether the speechis shouted or conversational.

The processing unit may apply various well known audio analysistechniques, or other techniques now known or later developed, todetermine the one or more linguistic characteristics in the audio inputand may do so in real-time (e.g., continually or periodically as thehearing prosthesis receives the audio input).

For example, the processing unit may apply various well known trainableclassifier techniques, such as neural networks, Gaussian Mixture models,Hidden Markov models, and tree classifiers. These techniques can betrained to recognize particular linguistic characteristics such as someof those noted above. For instance, a tree classifier can be used todetermine the presence of speech in audio input. Further, various onesof these techniques can be trained to recognize segments or quiet spacesbetween words, and to recognize the difference between male and femalevoices. Moreover, these techniques could be scaled in order ofcomplexity based on the extent of available computation power.

Implementation of a classifier may require several stages of processing.In a two-stage classifier, for instance, the first stage is used toextract information from a raw signal representing the received audioinput provided by the one or more microphones. This information can beanything from the raw audio signal itself, to specific features of theaudio signal (“feature extraction”), such as pitch, modulation depthetc. The second stage then uses this information to identify one or moreprobability estimates for a current class at issue.

In order for the second stage of this technique to work, it should betrained. Training involves, by way of example, collecting a pre-recordedset of example outputs (“training data”) from the system to beclassified, representing what engineers or others agree is a highestprobability classification from a closed set of possible classes to beclassified, such as audio of music or speech recorded through theprosthesis microphones. To train the second stage, this training data isthen processed by the first stage feature extraction methods, and thesefirst stage features are noted and matched to the agreed class. Throughthis design process, a pattern will ultimately be evident among all thefeature values versus the agreed class collected. For example, all ofthe speech samples might have a modulation depth above 0.5, while allnoise signals might be below 0.2. Well-known algorithms may then beapplied to help sort this data and to decide how best to implement thesecond stage classifier using the feature extraction and training dataavailable. For example, in a tree classifier, a decision tree may beused to implement an efficient method for the second stage. Nodes of thetree may thus have values such as “is modulation depth <0.5” asconditions for which direction to branch. And each path may end at ahighest probability class decision (such as a classification as music,speech, etc.)

In applying such a technique to identify linguistic characteristics inthe received audio input, the training data may for example containspoken words and sentences, by male and female speakers of various ages,and perhaps speech specifically by the recipient. Further, the featureextraction stage may contain voiced and unvoiced speech segmentdetectors, and perhaps a fast moving level measure to track the timebetween gaps in sentences. A two-stage classifier could then be trainedto recognize when a sentence had been spoken, and to distinguish othersounds as not being a sentence.

As another example, the processing unit may apply threshold detectors,such as to detect volume and energy levels in the received audio input,and to detect segments that indicate spaces between words, and perhapsto thereby detect the presence of words and sentences. Similar to thefirst stage of a classifier as discussed above, for instance, a set ofsignal features could be used to extract information from the audioinput signal, and the processing unit may forego applying the secondstage for classification. For example, the processing unit may detectthreshold modulation depth in the audio input as an indication of thepresence of speech, as speech may have modulation greater than othersignals carrying little information (such as noise). Further, theprocessing unit may detect threshold root mean squared (RMS) signallevel as a measure of gaps between sentences, to facilitate determininga quantity of sentences spoken.

By setting an appropriate threshold or thresholds for each of varioussignal features, the processing unit may thereby detect particularlinguistic features. For example, if the modulation depth is greaterthan 0.5 and the RMS level is above 60 dB SPL, the processing unit mayprogrammatically conclude that speech is present. Alternatively if themodulation depth is below 0.5 and the RMS level is below 50 dB SPL, theprocessing unit may determine that no speech is present. Further, bycounting the length and difference between speech and non-speechsegments, the processing unit may determine the number of sentencesspoken.

As still another example, the processing unit may apply various wellknown speech recognition techniques to detect the extent of speech inthe audio input. Those techniques may require significant computationalpower and may or may not be suitable for real-time analysis byprosthesis processing units without the assistance of an externalprocessing unit for instance. However, continued developments insignaling processing technology and speech recognition algorithms maymake actual speech recognition, including speaker recognition, moresuitable for implementation by the processing unit of a hearingprosthesis.

Further, in terms of determining whether identified speech is speech ofthe recipient or speech of another person in the recipient'senvironment, the processing unit may take various factors intoconsideration. For instance, the processing unit may take into accountloudness and frequency range of the speech, possibly by way ofcomparison with test samples of the recipient's own voice. In addition,if the prosthesis has multiple microphones for receiving the audio inputand the processing unit receives separate audio input from eachmicrophone, the processing unit may use those separate inputs todifferentiate between (i) the recipient's speech as may be picked up bya microphone positioned to best pick up speech coming from the recipientand (ii) others' speech as may be picked up by a microphone positionedto best pick up speech directed at the recipient.

Moreover, to facilitate carrying out this analysis in real-time, theprocessing unit may limit its analysis to identify key parameters asproxies for more complex linguistic characteristics or may generallyestimate various ones of the linguistic characteristics rather thanstriving to determine them exactly. For instance, rather than working todetermine an exact count of words spoken by the recipient or spoke byothers in the recipient's environment, the processing unit may determinean approximate count. Such an approximation may be clinically relevant,as it may facilitate general comparisons between extents of speech towhich the recipient is exposed. For example, if the processing unitdetermines that the recipient is exposed to approximately 400 words oneday and approximately 600 words the next day, that 50% estimatedincrease may be key to evaluating the recipient's speech exposure.

Optimally, as the processing unit receives the audio input, theprocessing unit may record various associated data in data storage 22.Further, the processing unit may output the data in real-time or at somelater time to the user interface system 24.

By way of example, as the processing unit determines the one or morelinguistic characteristics of the recipient's audio environment, theprocessing unit may record those characteristics in correspondence withindications of whether the hearing prosthesis is in the stimulation-onmode or is rather in the stimulation-off mode. For instance, theprocessing unit may keep track over time of the rate or number of words,sentences, or the like, in the audio input at times when the prosthesisis in the stimulation-on mode and may separately keep track over time ofthe rate or number of words, sentences, or the like, in the audio inputat times when the prosthesis is in the stimulation-off mode. And theprocessing unit may output data representing these metrics, possibly inreal-time as the processing unit generates the metrics. For instance, asnoted above, the user interface system 24 may periodically poll theprosthesis for such metrics and may receive and timestamp the metrics,to facilitate determining and presenting changes in the metrics overtime. Alternatively, the processing unit may push the data periodicallyto the user interface system or otherwise output the data.

Alternatively, the processing unit may limit its determination andlogging of linguistic characteristics to be just with respect to audioinput that the prosthesis receives at times when the hearing prosthesisis in the stimulation-on mode. For instance, the processing may trackthe linguistic characteristics just at times when the prosthesis is inthe stimulation-on mode, and not at times when the prosthesis is in thestimulation off mode. And the processing unit may output datarepresenting the determined linguistic characteristics, possibly inreal-time as the processing unit determines the linguisticcharacteristics.

As another example, as the processing unit receives the audio input, theprocessing unit may record the digitized audio itself, or arepresentation of the digitized audio, in correspondence withindications of whether the hearing prosthesis is in the stimulation-onmode or is rather in the stimulation off mode, with or withoutdetermining one or more linguistic characteristics in the audio input.For instance, using an internal or real-world clock, the processing unitmay timestamp the digitized audio (e.g., with periodic timestamps) andmay correspondingly record times when the prosthesis transitions betweenstimulation-on mode to stimulation-off mode, and the combination ofthose two sets of data would thus indicate which portions of the audioinput were received by the prosthesis when the prosthesis was in thestimulation-on mode and which portions of the audio input were receivedby the prosthesis when the prosthesis was in the stimulation-off mode.And the processing unit may output this data as well, again possibly inreal-time as the processing unit establishes this data.

As still another example, in regular operation, the processing unit maybe configured to record assorted other data related to operation of thehearing prosthesis, again in correspondence with indications of when theprosthesis is in the stimulation-on mode and when the prosthesis is inthe stimulation-off mode.

For instance, the processing unit may record instances of the prosthesisreceiving certain control signals from the user interface system 24,such as instances of user input changing various programs or otheroperational parameters of the prosthesis, and the processing unit maycorrelate those recorded instances with indications of the stimulationmode. Likewise, the processing unit may record various signal processingparameters of the processing unit, such as parameters of one or moreclassifier algorithms used to determine linguistic characteristicsand/or parameters used by the translation module 18 to generatestimulation signals based on the received audio input.

As with the linguistic characteristics, the processing unit may keeptrack of such operational parameters and changes in operationalparameters at times when the prosthesis is in the stimulation-on modeand may separately keep track of such operational parameters and changesin operational parameters at times when the prosthesis is in thestimulation-off mode. Furthermore, the processing unit may additionallyinclude with this data various determined linguistic characteristics asdiscussed above. And here again, the processing unit may output thisdata, possibly in real-time as the processing unit establishes the data.

In practice, the processing unit may provide this and other data to theuser interface system 24 in various forms for presentation to a usersuch as the recipient or a clinician. For example, the processing unitmay provide the data in raw form, as one or more lists of metrics andassociated values, such as a list of metrics corresponding withstimulation-on mode and a separate list of metrics corresponding withstimulation-off mode. As another example, the processing unit maystructure the data as graphs and other charts more readilyunderstandable at quick glance. For instance, rather than or in additionto listing the number of words spoken by the recipient on each ofvarious days, the processing unit may provide a graph that shows changein number of words spoke per day or per other unit of time, which couldthen be analyzed in terms of the recipient's environment. In practice,the processing unit may generate these graphs as graphical userinterfaces suitable for presentation by display 26.

In an alternative arrangement, note also that some of this analysis andpresentation could be done by an external processing unit, such asprocessing unit 30 of an external computing device. In line with thediscussion above, for instance, the processing unit 16 of the hearingprosthesis may record separate sets of linguistic characteristicscorresponding with stimulation-on mode and stimulation-off mode, andprocessing unit 16 may periodically or otherwise from time to timeprovide the latest such sets of data to the processing unit 30 of theexternal computing device. Upon receipt of such data, processing unit 30may then timestamp each received set of data with an indication of thecurrent day, time of day, or the like. And processing unit 30 of theexternal computing device may then analyze the data to determine one ormore linguistic characteristics in the audio, again possibly incorrespondence with the stimulation mode of the prosthesis, and maysimilarly present output representing that information, such adepictions of changes in linguistic characteristics in the recipient'saudio environment over time.

FIG. 2 is next a block diagram depicting more specifically variouscomponents that may be included in a representative processing unit 16and user interface system 24 in accordance with the present disclosure.In particular, FIG. 2 depicts processing unit 16 as a sound processorand user interface system 24 as a real-time monitor, such as a PC,smartphone, and/or remote control. The figure depicts on the soundprocessor a representative signal processing path for core hearingtherapy. Further, the figure depicts extraction of certain metrics fromvarious signal processing blocks, and forwarding of those metrics to alogging engine. The logging engine may then function to categorize themetrics, establish linguistic characterizations, and log thecharacterizations such as by incrementing counts of particularlinguistic characterizations (e.g., number of words spoken by therecipient, number of words spoken by others, etc.), in correspondencewith stimulation mode as discussed above. And the real-time monitor isthen connected to the sound processor so as to read the stored logs,such as by periodically polling for the latest logged data. And theauxiliary device may timestamp and that data for comparison andtrending, such as to determine and present indications of changes overtime (e.g., one week versus the last, one month versus the last, etc.)in linguistic characteristics in the recipient's environment.

FIG. 3 is next a flow chart depicting functions that can be carried outin accordance with a representative method. As shown in FIG. 3, at block34, the method includes receiving audio input into the hearingprosthesis 12 that is operable to stimulate a physiological system of arecipient in accordance with the received audio input, the receivedaudio input representing an audio environment of the recipient. Further,at block 36, the method involves the hearing prosthesis determining,based on the received audio input, one or more linguisticcharacteristics of the audio environment. Still further, at block 38,the method involves the hearing prosthesis generating data representingthe one or more determined linguistic characteristics. And at block 40,the method involves outputting the data from the hearing prosthesis,such as providing the data to a computing system for presentation.

Although these functions are shown in series in the flow chart, thehearing prosthesis may in practice continuously carry out thesefunctions in real-time. For instance, as the hearing prosthesis receivesaudio input, the hearing prosthesis may continuously or periodicallyanalyze the audio input to determine linguistic characteristics and maycontinuously or periodically generate and output data representing thelatest determined linguistic characteristics.

In line with the discussion above, the hearing prosthesis may be ahearing aid worn by the recipient or a device at least partiallyimplanted in the recipient, and the act of stimulating the physiologicalsystem of the recipient may accordingly take various forms. For example,the hearing prosthesis could be a hearing aid, in which case the act ofstimulating the physiological system of the recipient may involvedelivering into an ear of the recipient an amplification of the receivedaudio input. Alternatively, the hearing prosthesis could be a partiallyor fully implanted cochlear implant, in which case the act ofstimulating the physiological system of the recipient may involvestimulating one or more cochlear electrodes of the recipient inaccordance with the received audio input. And still alternatively, thehearing prosthesis could be a vibrationally-coupled hearing prosthesis,in which case the act of stimulating the physiological system of therecipient may involve delivering vibrations to the recipient inaccordance with the received audio input. Other examples are possible aswell.

Further, in line with the discussion above, the hearing prosthesis mayhave a stimulation mode that switches between a stimulation-on mode inwhich the hearing prosthesis is set to stimulate the physiologicalsystem of the recipient in accordance with the received audio input anda stimulation-off mode in which the hearing prosthesis is set to notstimulate the physiological system of the recipient in accordance withthe received audio input. And in that case, the hearing prosthesis maytake the stimulation mode into account while generating the datarepresenting the determined one or more linguistic characteristics.

For example, the hearing prosthesis may include in the data indicia ofwhen the hearing prosthesis was in the stimulation-on mode and/or whenthe hearing prosthesis was in the stimulation-mode. More particularly,the hearing prosthesis may include in the data one or more correlationsbetween times when the hearing prosthesis was in the stimulation-on modeand one or more of the one or more determined linguisticcharacteristics, and perhaps similarly for times when the hearingprosthesis was in the stimulation-off mode. Similarly, the hearingprosthesis may maintain separate sets of data for each stimulation mode.For instance, separately for stimulation-on mode and stimulation-offmode, the hearing prosthesis may maintain counters or other rolled upindications of various determined linguistic characteristics, and thehearing prosthesis may increment or otherwise adjust those indicationsof determined linguistic metrics. The hearing prosthesis may then outputthose separate sets of data to facilitate separate analysis linguisticcharacteristics present in the recipient's environment during times ofthe prosthesis being in stimulation-on mode and linguisticcharacteristics present in the recipient's environment during times ofthe prosthesis being in stimulation-off mode.

As another example, using an internal clock or a real-world clock, thehearing prosthesis may time-stamp the received audio input and mayrecord times when the hearing prosthesis was in the stimulation-on modeand times when the hearing prosthesis was in the stimulation-off mode.And the hearing prosthesis may correlate the recorded times with thetimestamped received audio input so as to determine which audio inputwas received during stimulation-on mode and which audio input wasreceived during stimulation-off mode. With that information, the hearingprosthesis may then provide output correlating certain determinedlinguistic characteristics with stimulation-on mode and other determinedlinguistic characteristics with stimulation-off mode.

Alternatively or additionally, the hearing prosthesis may determine theone or more linguistic characteristics specifically based on the audioinput received during stimulation-on mode and not based on audio inputreceived during stimulation-off mode. Further, whether or not thehearing prosthesis determines one or more linguistic characteristicsbased on the audio input received during stimulation-off mode, thehearing prosthesis could be arranged to limit the generated data to bebased on just the audio input received during stimulation-on mode.

In practice as discussed above, the one or more linguisticcharacteristics may include a quantity of speech, such as a measure ofquantity of speech by the recipient and a measure of quantity of speechby one or more people other than the recipient. Further, the one or morelinguistic characteristics may comprise any of those noted above orother characteristics that are specifically related to languageproduction and/or language receipt. And any or all of the determinedlinguistic characteristics may include a measure that is an estimate.

Still further, as discussed above, the act of receiving the audio inputmay involve receiving the audio input concurrently from multiplemicrophones. And in that case, the hearing prosthesis may determinebased at least in part on the audio input received by the multiplemicrophones whether the audio input represents speech by the recipientor rather represents speech by a person other than the recipient.

In line with the discussion above, the data output by the hearingprosthesis in this method may be used to develop speech training for therecipient. For instance, given data indicating that the recipient isbeing exposed to a certain extent of speech (including speechproduction, speech reception, conversations, and so forth), a clinicianmay arrange for the recipient to be exposed to more speech, or to speechof different types, in an effort to help rehabilitate the recipient.Further, for recipients who are initially developing their speech skills(such as infants or the like), a clinician could use this output data tohelp identify speech errors, such as phoneme substitution for instance,and to develop appropriate therapy and further speech training.

FIG. 4 is next another flow chart depicting functions that can becarried out in accordance with a representative method. As shown in FIG.4, at block 42, the method involves receiving audio input into a hearingprosthesis that is operable to stimulate a physiological system of arecipient in accordance with the received audio input, where at timeswhile receiving the audio input the hearing prosthesis is in astimulation-on mode in which it is set to stimulate the physiologicalsystem of the recipient in accordance with the received audio input andat other times while receiving the audio input the hearing prosthesis isin a stimulation-off mode in which it is set to not stimulate thephysiological system of the recipient in accordance with the receivedaudio input. Further, at block 44, the method involves the hearingprosthesis logging data representing the received audio input incorrespondence with the times when the hearing prosthesis is in thestimulation-on mode, such as recording in or in correlation with thelogged data indicia of when the hearing prosthesis is in thestimulation-on mode, and perhaps separately when the hearing prosthesisis in the stimulation-off mode, or by maintaining counters separatelyfor the two stimulation modes.

FIG. 5 is next yet another flow chart depicting functions that can becarried out in accordance with a representative method. As shown in FIG.5, at block 50, the method involves receiving audio input into a hearingassistance device that is worn by or at least partially implanted in ahuman recipient and that is operable to stimulate a physiological systemof the recipient in accordance with the received audio input, thereceived audio input representing an audio environment of the recipient.Further, at block 52, the method involves the hearing assistance devicerecording data representing the received audio input and specifyingtimes when the hearing assistance device was operating in astimulation-on mode in which the hearing assistance device was set tostimulate the physiological system of the recipient in accordance withthe received audio input. Still further, at block 54, the methodinvolves determining based on the recorded data one or more linguisticcharacteristics of the audio environment. And at block 56, the methodinvolves providing output representing the one or more determinedlinguistic characteristics.

In practice, as discussed above, the hearing assistance device itselfmay carry out the acts of determining the one or more linguisticcharacteristics and providing the output representing the one or moredetermined linguistic characteristics. Alternatively, a processingsystem external to the hearing assistance device, such as a separatecomputing device for instance, may carry out those functions, in whichcase the method may further include transferring the data from thehearing assistance device to the external processing system. In eithercase, as further discussed above, the output may also associate the oneor more determined linguistic characteristics with the specified timeswhen the audio input caused the hearing assistance device to stimulatethe physiological system of the recipient.

Exemplary embodiments have been described above. It should beunderstood, however, that numerous variations from the embodimentsdiscussed are possible, while remaining within the scope of theinvention.

What is claimed is:
 1. A hearing prosthesis, comprising: one or moremicrophones configured to receive audio inputs representing an audioenvironment of the hearing prosthesis, wherein the hearing prosthesis isconfigured to selectively stimulate a physiological system of arecipient in accordance with one or more of the audio inputs; one ormore stimulus outputs; a processing unit configured to: determine, basedon the audio inputs, one or more linguistic characteristics of the audioenvironment, generate data representing the one or more linguisticcharacteristics; and output the data from the hearing prosthesis.
 2. Thehearing prosthesis of claim 1, wherein the one or more linguisticcharacteristics comprise a measure of speech produced by the recipient.3. The hearing prosthesis of claim 2, wherein the measure of speechproduced by the recipient comprises a measure of quantity of speech bythe recipient.
 4. The hearing prosthesis of claim 2, wherein the measureof speech produced by the recipient comprises at least onecharacteristic selected from the group consisting of: a measure ofproportion of time spent by the recipient speaking, a measure ofquantity of conversational turns by the recipient, a measure of quantityof conversations initiated by the recipient, a measure of quantity ofphonetic features produced by the recipient, and a measure of length ofutterances by the recipient.
 5. The hearing prosthesis of claim 1,wherein the one or more linguistic characteristics comprise a measure ofspeech produced by one or more people other than the recipient.
 6. Thehearing prosthesis of claim 5, wherein the measure of speech produced byone or more people other than the recipient comprises a measure ofquantity of speech produced by the one or more people other than therecipient.
 7. The hearing prosthesis of claim 1, wherein the one or morelinguistic characteristics comprise a measure of quantity of speechinteraction between the recipient and one or more people other than therecipient.
 8. The hearing prosthesis of claim 1, wherein the one or morelinguistic characteristics comprise a measure of speech quality.
 9. Thehearing prosthesis of claim 1, wherein the linguistic characteristicscomprise (i) a measure of quantity of speech by the recipient and (ii) ameasure of quantity of speech by one or more people other than therecipient.
 10. The hearing prosthesis of claim 1, wherein the hearingprosthesis is configured to selectively operate in a stimulation-on modein which the processing unit is configured to convert the audio inputsinto stimulation signals for delivery to the physiological system of therecipient via the one or more stimulus outputs and a stimulation-offmode in which the processing unit does not convert the audio inputs intostimulation signals.
 11. The hearing prosthesis of claim 10, wherein theprocessing unit only generates the data representing the one or morelinguistic characteristics when the hearing prosthesis is in thestimulation-on mode.
 12. The hearing prosthesis of claim 10, wherein togenerate data representing the one or more determined linguisticcharacteristics, the processing unit is configured to: log the one ormore linguistic characteristics separately for (i) when the hearingprosthesis is in the stimulation-on mode and (ii) when the hearingprosthesis is in the stimulation-off mode.
 13. The hearing prosthesis ofclaim 12, wherein the data output by the hearing prosthesis associatesthe one or more linguistic characteristics with specified times when thehearing prosthesis was operating in the stimulation-on mode.
 14. Thehearing prosthesis of claim 10, wherein to output the data from thehearing prosthesis, the processing unit is configured to: outputseparate sets of data (i) corresponding with when the hearing prosthesiswas in the stimulation-on mode and (ii) corresponding with when thehearing prosthesis was in the stimulation-off mode.
 15. The hearingprosthesis of claim 10, wherein the hearing prosthesis is switchablebetween the stimulation-on mode and the stimulation-off mode by therecipient.
 16. The hearing prosthesis of claim 1, wherein the hearingprosthesis comprises a hearing aid and wherein the one or more stimulusoutputs comprise a receiver configured to output an amplified acousticsignal.
 17. The hearing prosthesis of claim 1, wherein the hearingprosthesis comprises a cochlear implant, and wherein the one or morestimulus outputs comprise one or more cochlear electrodes configured todeliver electrical stimulation signals to the recipient.
 18. The hearingprosthesis of claim 1, wherein the hearing prosthesis comprises avibrationally-coupled hearing prosthesis configured to delivervibrations to the recipient in accordance with the audio inputs.
 19. Thehearing prosthesis of claim 1, wherein the processing unit is configuredto: determine the one or more linguistic characteristics of the audioenvironment, generate the data representing the one or more linguisticcharacteristics, and output the data in real-time as the hearingprosthesis receives the audio input.
 20. The hearing prosthesis of claim1, wherein the processing unit is configured to output the data from thehearing prosthesis to a computing system for presentation to a user. 21.The hearing prosthesis of claim 1, wherein the hearing prosthesisreceives a plurality of audio inputs, and wherein the processing unit isconfigured to: record times when one or more of the plurality of audioinputs causes the hearing prosthesis to stimulate the physiologicalsystem of the recipient; use the recorded times to determine a selectedone or more of the plurality of audio inputs that caused the hearingprosthesis to stimulate a physiological system of the recipient; anddetermine the linguistic characteristics based only the one or more ofthe plurality of audio inputs.
 22. The hearing prosthesis of claim 1,further comprising: a plurality of microphones configured to receive theaudio inputs, and wherein the processing unit is configured to:determine based at least in part on audio inputs received by theplurality of microphones whether the audio inputs represent speech bythe recipient or rather represent speech by a person other than therecipient.